Models with incorrect tokenizer_class in tokenization_config.json tha…#44179
Models with incorrect tokenizer_class in tokenization_config.json tha…#44179ArthurZucker merged 1 commit intomainfrom
Conversation
…t should use TokenziersBackend
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: auto, deepseek_vl, deepseek_vl_hybrid, jamba, janus, llava, llava_next, phi3, vipllava |
|
This comment contains models: ["models/auto", "models/deepseek_vl", "models/deepseek_vl_hybrid", "models/jamba", "models/janus", "models/llava", "models/llava_next", "models/phi3", "models/vipllava"] |
ArthurZucker
left a comment
There was a problem hiding this comment.
would like to have a super minimal list but yes otherwise
| "llava", | ||
| "llava_next", |
There was a problem hiding this comment.
are these from the vllm ci as well?
CI ResultsCommit Info
Model CI Report❌ 7 new failed tests from this PR 😭
|
|
the batched failures predate the bad commit and aren't related to tokenizers I believe 😢 https://huggingface.co/datasets/hf-internal-testing/transformers_daily_ci/raw/e7e22bff009cb47fe2dea7802813b455c53f4816/2026-02-19/ci_results_run_models_gpu/new_failures_with_bad_commit_grouped_by_authors.json |
Models with incorrect tokenizer_class in tokenization_config.json that should use TokenziersBackend